[AMD] [AgentX] Add MI355X Qwen3.5 FP4 SGLang MTP - #2562
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献!请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人,你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板,包括保留英文语句 如需进行 PR 验证,请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动(flake),重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
1 similar comment
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献!请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人,你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板,包括保留英文语句 如需进行 PR 验证,请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动(flake),重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
There was a problem hiding this comment.
An inline bug was found that will break the throughput sweep for this recipe, so this needs a human look and a fix before merge.
Extended reasoning...
This PR is a new benchmark recipe (script + config + changelog entry) for an existing model/hardware pattern, but the inline finding is a real, run-breaking bug: appending --use-chat-template to REPLAY_CMD after build_replay_cmd will make aiperf profile reject the flag and fail every throughput concurrency point in this recipe.
| build_replay_cmd "$RESULT_DIR" | ||
| REPLAY_CMD+=" --use-chat-template" | ||
| run_agentic_replay_and_write_outputs "$RESULT_DIR" | ||
| fi |
There was a problem hiding this comment.
🔴 Line 177 appends REPLAY_CMD+=" --use-chat-template" after build_replay_cmd, but build_replay_cmd (benchmark_lib.sh:1927) builds an aiperf profile invocation for the agentic /v1/chat/completions replay path, not the run_benchmark_serving (benchmark_serving.py) path where this flag is actually recognized. This will make aiperf profile reject the unrecognized argument and exit non-zero, failing every non-eval (throughput) concurrency point in this new recipe. Fix: delete line 177 entirely.
Extended reasoning...
build_replay_cmd in benchmark_lib.sh:1887-2053 constructs REPLAY_CMD as an $AIPERF_CLI profile --scenario inferencex-agentx-mvp ... invocation, targeting --endpoint /v1/chat/completions with --endpoint-type chat. Nowhere in this function (or anywhere in the aiperf profile argument set) is --use-chat-template handled — that flag is defined and consumed exclusively inside run_benchmark_serving (benchmark_lib.sh:520 for parsing, :636-638 for application), which drives utils/bench_serving/benchmark_serving.py for the fixed-seq-len scripts, not the agentic replay path.
In qwen3.5_fp4_mi355x_sglang_mtp.sh:176-179, the non-EVAL_ONLY branch calls build_replay_cmd "$RESULT_DIR" and then appends REPLAY_CMD+=" --use-chat-template" before calling run_agentic_replay_and_write_outputs "$RESULT_DIR". That function executes $REPLAY_CMD literally at benchmark_lib.sh:2112 ($REPLAY_CMD 2>&1 | tee "$result_dir/benchmark.log") and gates success on the exit code (replay_rc). Since aiperf profile has no such flag, it will reject the unrecognized CLI argument and exit non-zero, causing run_agentic_replay_and_write_outputs to treat the run as failed.
This is not a stylistic mismatch — it's a straightforward "passing an unsupported flag to a CLI tool" bug. The only reason it isn't obviously wrong from a shallow read is that --use-chat-template is a real, valid flag elsewhere in the same file, which makes it plausible at a glance that it belongs here too. But tracing which command REPLAY_CMD actually becomes shows it doesn't apply to this call site.
Corroborating evidence from the rest of the codebase: (1) the direct sibling B200 recipe qwen3.5_fp4_b200_sglang_mtp.sh appends --server-metrics after build_replay_cmd, never --use-chat-template; (2) no other agentic *_mtp.sh script (dsv4 variants, glm5.2, etc.) adds this flag to REPLAY_CMD; (3) kimik3_fp4_b300_vllm_mtp.sh:35-38 has an explicit comment stating exactly why agentic recipes must not add it: "AGENTS.md requires MTP scripts to pass --use-chat-template to run_benchmark_serving. Agentic recipes never call it -- the replay drives AIPerf against /v1/chat/completions, so prompts are already chat-formatted ... Nothing to add here."
Step-by-step proof of the failure: (1) Script reaches the else branch (EVAL_ONLY defaults to false, so this is the default/majority path — the sweep's agentic-coding scenario in configs/amd-master.yaml runs many conc-list throughput points, e.g. conc-list: [1, 4, 8, 12, 16, ...]). (2) build_replay_cmd "$RESULT_DIR" sets REPLAY_CMD="$AIPERF_CLI profile --scenario inferencex-agentx-mvp --url http://localhost:$PORT --endpoint /v1/chat/completions --endpoint-type chat ... --output-artifact-dir $RESULT_DIR/aiperf_artifacts ...". (3) Line 177 appends --use-chat-template to this string. (4) run_agentic_replay_and_write_outputs runs $REPLAY_CMD, i.e. invokes aiperf profile ... --use-chat-template. (5) aiperf's CLI parser (argparse or similar) has no such option registered for the profile subcommand and will error out ("unrecognized arguments: --use-chat-template") with a non-zero exit code. (6) replay_rc becomes non-zero, and the run is recorded as failed — for every throughput concurrency point across all four search-space rows in the new qwen3.5-fp4-mi355x-sglang-agentic-mtp config entry.
The fix is simply to delete line 177 (REPLAY_CMD+=" --use-chat-template"), leaving build_replay_cmd's output untouched, matching every other agentic *_mtp.sh script's pattern.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31544080186 |
Add an SGLang EAGLE AgentX recipe, B200/B300-parity TP4 and TP2 discovery coverage, MI355X HiCache tiers, and required server metrics. Use SGLang v0.5.17 and disable unstable AITER all-reduce fusion to preserve TP2/EP2 rank consistency.
ff3fa97 to
7453c5e
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31581562219 |
|
/reuse-sweep-run 31581562219 |
|
/stage-results 31581562219 |
|
@cquil11 staged run 31581562219: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-08-12~r31581562219 This run remains available across future |
|
/reuse-sweep-run 31581562219 |
Summary
Add MI355X Qwen3.5-397B-A17B MXFP4 AgentX with SGLang native EAGLE MTP.
Performance scope
Validation
Next: full-duration official sweep on the retained matrix.